from sklearn_benchmarks.reporting.hp_match import HpMatchReporting
import pandas as pd
pd.set_option('display.max_colwidth', None)
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)

scikit-learn vs. ONNX Runtime

reporting = HpMatchReporting(other_library="onnx", config="config.yml", log_scale=False)
reporting.make_report()

We assume here there is a perfect match between the hyperparameters of both librairies. For a given set of parameters and a given dataset, we compute the speedup time scikit-learn / time onnx. For instance, a speedup of 2 means that onnx is twice as fast as scikit-learn for a given set of parameters and a given dataset.

KNeighborsClassifier_brute_force

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=brute.

Raw results

predict

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.154	0.152	0.000	0.002	-1	1	0.663	20.208	0.160	0.663	0.107	0.107	See	See
1	KNeighborsClassifier_brute_force	predict	100000	1	100	0.027	0.002	0.000	0.027	-1	1	1.000	0.367	0.003	1.000	0.073	0.073	See	See
2	KNeighborsClassifier_brute_force	predict	100000	1000	100	3.288	0.092	0.000	0.003	-1	5	0.757	20.187	0.202	0.757	0.163	0.163	See	See
3	KNeighborsClassifier_brute_force	predict	100000	1	100	0.028	0.004	0.000	0.028	-1	5	1.000	0.367	0.006	1.000	0.076	0.076	See	See
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.300	0.013	0.000	0.002	1	100	0.882	20.115	0.068	0.882	0.114	0.114	See	See
5	KNeighborsClassifier_brute_force	predict	100000	1	100	0.023	0.000	0.000	0.023	1	100	1.000	0.373	0.006	1.000	0.062	0.062	See	See
6	KNeighborsClassifier_brute_force	predict	100000	1000	100	3.071	0.034	0.000	0.003	-1	100	0.882	20.410	0.024	0.882	0.150	0.150	See	See
7	KNeighborsClassifier_brute_force	predict	100000	1	100	0.027	0.003	0.000	0.027	-1	100	1.000	0.366	0.006	1.000	0.073	0.073	See	See
8	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.310	0.007	0.000	0.002	1	5	0.757	20.091	0.070	0.757	0.115	0.115	See	See
9	KNeighborsClassifier_brute_force	predict	100000	1	100	0.024	0.000	0.000	0.024	1	5	1.000	0.368	0.006	1.000	0.066	0.066	See	See
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.253	0.003	0.001	0.001	1	1	0.663	20.106	0.080	0.663	0.062	0.062	See	See
11	KNeighborsClassifier_brute_force	predict	100000	1	100	0.023	0.001	0.000	0.023	1	1	1.000	0.369	0.006	1.000	0.062	0.062	See	See
12	KNeighborsClassifier_brute_force	predict	100000	1000	2	1.795	0.029	0.000	0.002	-1	1	0.896	4.326	0.009	0.896	0.415	0.415	See	See
13	KNeighborsClassifier_brute_force	predict	100000	1	2	0.006	0.004	0.000	0.006	-1	1	1.000	0.286	0.005	1.000	0.021	0.021	See	See
14	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.909	0.043	0.000	0.003	-1	5	0.922	4.257	0.010	0.922	0.683	0.683	See	See
15	KNeighborsClassifier_brute_force	predict	100000	1	2	0.013	0.004	0.000	0.013	-1	5	1.000	0.284	0.004	1.000	0.046	0.046	See	See
16	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.185	0.003	0.000	0.002	1	100	0.929	4.307	0.005	0.929	0.507	0.507	See	See
17	KNeighborsClassifier_brute_force	predict	100000	1	2	0.004	0.000	0.000	0.004	1	100	1.000	0.286	0.005	1.000	0.013	0.013	See	See
18	KNeighborsClassifier_brute_force	predict	100000	1000	2	3.162	0.094	0.000	0.003	-1	100	0.929	4.309	0.007	0.929	0.734	0.734	See	See
19	KNeighborsClassifier_brute_force	predict	100000	1	2	0.010	0.005	0.000	0.010	-1	100	1.000	0.285	0.004	1.000	0.033	0.033	See	See
20	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.194	0.010	0.000	0.002	1	5	0.922	4.252	0.004	0.922	0.516	0.516	See	See
21	KNeighborsClassifier_brute_force	predict	100000	1	2	0.004	0.000	0.000	0.004	1	5	1.000	0.282	0.003	1.000	0.013	0.013	See	See
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	1.103	0.006	0.000	0.001	1	1	0.896	4.328	0.010	0.896	0.255	0.255	See	See
23	KNeighborsClassifier_brute_force	predict	100000	1	2	0.002	0.000	0.000	0.002	1	1	1.000	0.282	0.002	1.000	0.008	0.008	See	See

KNeighborsClassifier_kd_tree

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=kd_tree.

Raw results

predict

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.862	1.047	0.000	0.001	-1	1	0.929	123.905	0.000	0.929	0.007	0.007	See	See
1	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	0.000	0.003	-1	1	1.000	3.162	0.260	1.000	0.001	0.001	See	See
2	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.141	0.413	0.000	0.001	-1	5	0.946	121.483	0.000	0.946	0.009	0.009	See	See
3	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.004	0.000	0.000	0.004	-1	5	1.000	3.116	0.210	1.000	0.001	0.001	See	See
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	5.865	0.463	0.000	0.006	1	100	0.951	122.601	0.000	0.951	0.048	0.048	See	See
5	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.004	0.001	0.000	0.004	1	100	1.000	3.130	0.209	1.000	0.001	0.001	See	See
6	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	3.429	0.221	0.000	0.003	-1	100	0.951	122.596	0.000	0.951	0.028	0.028	See	See
7	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.007	0.001	0.000	0.007	-1	100	1.000	3.116	0.219	1.000	0.002	0.002	See	See
8	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.768	0.119	0.000	0.002	1	5	0.946	121.808	0.000	0.946	0.015	0.015	See	See
9	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.002	0.000	0.000	0.002	1	5	1.000	3.105	0.138	1.000	0.001	0.001	See	See
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.895	0.173	0.000	0.001	1	1	0.929	123.705	0.000	0.929	0.007	0.007	See	See
11	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.001	0.000	0.000	0.001	1	1	1.000	3.147	0.209	1.000	0.000	0.000	See	See
12	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.033	0.018	0.000	0.000	-1	1	0.891	0.052	0.016	0.891	0.622	0.650	See	See
13	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.003	0.001	0.000	0.003	-1	1	1.000	0.006	0.000	1.000	0.549	0.549	See	See
14	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.027	0.001	0.001	0.000	-1	5	0.911	0.055	0.018	0.911	0.492	0.518	See	See
15	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.000	0.000	0.002	-1	5	1.000	0.006	0.000	1.000	0.366	0.366	See	See
16	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.041	0.008	0.000	0.000	1	100	0.894	0.070	0.000	0.894	0.579	0.579	See	See
17	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.001	0.000	0.000	0.001	1	100	1.000	0.006	0.000	1.000	0.108	0.108	See	See
18	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.044	0.006	0.000	0.000	-1	100	0.894	0.070	0.000	0.894	0.625	0.625	See	See
19	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.003	0.002	0.000	0.003	-1	100	1.000	0.007	0.002	1.000	0.505	0.521	See	See
20	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.024	0.000	0.001	0.000	1	5	0.911	0.046	0.001	0.911	0.515	0.515	See	See
21	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.001	0.000	0.000	0.001	1	5	1.000	0.006	0.000	1.000	0.108	0.108	See	See
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.021	0.000	0.001	0.000	1	1	0.891	0.044	0.000	0.891	0.485	0.485	See	See
23	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.001	0.000	0.000	0.001	1	1	1.000	0.006	0.000	1.000	0.105	0.105	See	See

HistGradientBoostingClassifier_best

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: learning_rate=0.01, n_iter_no_change=10.0, max_leaf_nodes=100.0, max_bins=255.0, min_samples_leaf=100.0, max_iter=300.0.

Raw results

predict

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	HistGradientBoostingClassifier_best	predict	100000	1000	100	0.147	0.005	300	0.005	0.000	0.824	0.551	0.029	0.824	0.267	0.267	See	See
1	HistGradientBoostingClassifier_best	predict	100000	1	100	0.021	0.001	300	0.000	0.021	1.000	0.456	0.011	1.000	0.046	0.046	See	See